Topcoder Scorecards

Scorecards are used by reviewers to compute the quality of challenge submissions and the extent to which a submission meets the challenge specification requirements. Each challenge type will have a default scorecard, but make sure to view the scorecard before competing since it can change based on client, technology, or even coding styles requested!

Each scorecard has a set of questions or sections for which weights have been assigned. Depending on the score assigned for each question, the sum of the calculated weights then becomes the final score.

Each scorecard has a passing score associated with it. To be eligible for payments related to the contest, a submission must not only gain the first or second place (or other spots as applicable for the contest), but also a score greater than or equal to the passing score for that scorecard.

Current Version of the scorecard

Assembly and Code Scorecard

Writing a Good Challenge Specification

The scorecard exists to help get solid output from the challenge. It helps the process if you write your challenge in the format of the scorecard.

The current Code scorecard, for example, has these headings: Deployment Guide, Code Requirements, Contest Requirements, and Testing. In your challenge spec, if you list these headings and enumerate requirements for output under those headings, reviewers can basically use it as a checklist, listing defects referring to headings, and help streamline appeals

Performing Code Review

It is very important to perform thorough reviews of all submissions. Not only does code review determine the challenge winners and payouts, it produces better results for our clients as well. Furthermore, Topcoder members learn a ton from reviewing and being reviewed.

Smartbear (no relation to our own LazyBaer) has a great article on top 10 tips for performing a great code review. The whole list is worth reviewing but the three big items Topcoder wholeheartedly agrees with from their list are the following:

• Foster a positive code review culture - Everything at Topcoder is about our members. Reviews must be accurate, correct and direct but they will never be mean or confrontational. Everyone looks at a problem differently which is one of the great things about using a community to develop and even learn more about software development. Here are some tips on being constructive

  • Do not use confrontational terms like “this is not good” or "this is stupid’ or anything along those lines. Instead, use wording like “an alternative approach”, “one that has worked well before” or “previously Topcoder has done” are all more constructive ways to say the same thing.

  • Do not direct your comments at the member. Do not say “you” or “your code” is wrong but focus on the submission and or technology instead and try wording like “for this challenge this works better” or “for this framework/language it is recommended to”.

  • It is also ok to mix in some praise! If someone did something really cool, innovative or something that hasn’t been done before. Note it in the scorecard! A comment like “You did this really well” or “I haven’t seen this before, what a great use of JS” goes a long way to helping the member accept the criticism as well.

• Take your time - Reviewing code takes a bit of time. Our reviewers have to review multiple submissions. Try to space out your review time by using the save for later feature of the scorecard system. Topcoder recommends first deploying all the submissions (if applicable) and taking a break. Once you know the application is deployable, watch any of the attached videos before reviewing the application. We recommend taking a good 30 minute break at least between reviewing each submission. It is great to clear your head before coming back to do another review.

• Use a checklist - At Topcoder this is our scorecard. The scorecard enforces that all reviewers follow through the same process and we’ll go over that more in our next section.

Applying Scorecards During Review

Once the submission phase for a contest completes, the review phase begins. Here, reviewers evaluate the submissions and categorize their review comments based on the scorecard. Based on the comments and issues raised, the reviewer will accordingly assign scores to each section in the scorecard.

In case there are multiple reviewers, an aggregate score is calculated by averaging the reviewer scores together. Eventually, the aggregate score decides the submission’s ranking and placement with respect to other submissions.

The reviewers are allowed to provide two types of feedback - “Required” and “Comments”. Feedback marked as “Required” require the user to fix the issue raised (if there is a Final Fix phase or maybe fed into the requirements of subsequent challenges) and contribute to that section’s score. Feedback marked as “Comments” are simply points that the reviewer have brought up but do not affect the section score.

For example, let’s dive into our most recent scorecard (as of 8/24/16) as an example. Don’t forget to look at the weightings for each section and question. That will give you a sense of how important each section and question in each section is. We’ll walk through each section at a high level:

• Requirements – Here is our most IMPORTANT section. Topcoder’s work ultimately will be valued by the customer on the fact the code successfully implements the requirements. The review board ensures that all requirements are implemented and accounted. If any issues are identified here, the score will be marked down and a comment will be added describing the error.

  1. Code Requirements – All code product at Topcoder must follow best practices as defined at the start of the challenge. Best practices not only refer to following proper patterns and coding principles but also to proper commenting or documentation, cleaning up debug statements, and removing unused code. Keep in mind that not only should the code be usable today but it must be built for maintainability in the future as well.

  2. Deployment Guide – The deployment guide section ensures the code is deployable and testable. This section will score well if the code can be deployed following the steps provided and their is some demonstrative display of the application working. This last step can take many forms based on the challenges, a video, a test script output or assets produced from the code.

• Testing – Writing code is the riskiest thing you can do in a project. Each time code is touched there is a potential for bugs to creep into the code base! The easiest way to avoid and catch these bugs is to follow proper test development procedures. The questions in this section ensure that proper test development occurs as part of the Topcoder process.

Fielding Appeals

Once the review comments are in, participants can see the reviews. They are allowed to appeal against each score in a review. This is also captured in the scorecard. The reviewer can then respond to these appeals and the scores can be corrected based on the success of the appeal.

The final score at the end of the Appeals Response phase decides the placements and rankings and the winner of the contest.